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Program description 


Cognitive Tutor® Aigebra /, a full year course, delivers instruc- 
tion in single variable data, simplifying linear expressions, 
mathematical modeling, solving systems with linear equations, 
problem solving using proportional reasoning, and powers 
and exponents. Students work at their own pace to develop 


problem-solving skills. The duration of each lesson can vary, 
depending on the length of a school’s class period. Generally, 
three periods a week are spent using the Cognitive Tutor® 
Aigebra 1 text for classroom activities, and two are spent in the 
computer lab using the Cognitive Tutor® Aigebra 1 software. 


Research 


One study met the What Works Clearinghouse (WWC) evidence 
standards and one study met the WWC evidence standards 
with reservations. Together, the two studies included more than 
800 ninth graders in more than 40 classrooms in Florida and 


Cklahoma. The studies examined the effects of Cognitive Tutor® 
Aigebra 1 on students’ math achievement.^ 

The WWC considers the extent of evidence for Cognitive 
Tutor® Algebra 1 to be moderate to large for math achievement. 



Eff6CtiV6neSS Cognitive Tutor® Aigebra I was found to have potentially positive effects on math achievement. 



Math achievement 

Rating of effectiveness Potentially positive effects 
Improvement index^ Average: +8 percentile points 

Range: -1 to +16 percentile points 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average and range of improvement indices for all findings across the studies. 
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Updating previous report 

This report updates the previous WWC report on Cognitive 
Tutor® that was released on the WWC website in December 
2004. The report reieased in 2004 reviewed research on a variety 
of Cognitive Tutor® math programs. However, this report focuses 
on Cognitive Tutor® Aigebra I. 

Since the original review of Cognitive Tutor® was reieased in 
December 2004, the WWC has updated its evidence standards 
and developed peer-review procedures for adjusting such meth- 
odoiogicai flaws in studies as nonequivaient groups at pretest 
and a mismatch between the unit of assignment and the unit of 
anaiysis. These standards and procedures have been appiied to 
aii studies in this updated review. 

Developer and contact 

Distributed by Carnegie Learning, Inc. Address; Frick Buiiding, 
20th Fioor, 437 Grant Street, Pittsburgh, PA 15219. Emaii: info® 
carnegieiearning.com. Web: http://www.carnegieiearning.com. 
Teiephone: (888) 851-7094. 

Scope of use 

Pilot implementation of the curricuium began in 1992 with 84 
students in one school. As of December 2006, Cognitive Tutor® 
curricula, which inciude Bridge to Aigebra, Aigebra I, Aigebra II, 
Geometry, and Integrated Math, have been used by more than 
475,000 students in 1,300 urban, rurai, and suburban schooi 
districts across the United States. In 2006, Carnegie Learning 
revised some of the instructional and technological aspects 



of Cognitive Tutor® Aigebra I. According to the developer, no 
fundamental changes to the pedagogical approaches or content 
were made. 

Teaching 

Typically, three class periods a week are organized around 
textbook materials and small group activities. Teachers facilitate 
small group problem solving and whole classroom discussions. 
In the other two class periods, students work at their own pace 
to develop problem-solving skills by working on the computer 
with the Cognitive Tutor® Algebra I software. In the computer 
lab, teachers interact with students individually. 

Carnegie Learning provides a four-day preservice training. 
In-service professional development is also available during the 
year. Teacher training for Cognitive Tutor® Aigebra I (software and 
text) covers the philosophy and application of these products. 

The training sessions are conducted by Certified Implementation 
Specialists— current or former mathematics teachers who have 
completed in-depth training from Carnegie Learning’s staff of 
educators, technology specialists, and curriculum developers. 

Cost 

Cognitive Tutor® Aigebra I is offered to schools as annual site 
license configurations. According to the developer, pricing per 
student starts at $58.80 for the full curriculum— software, books, 
and maintenance. Volume and term discounts are available. 
Professional development costs $600 per teacher attending a 
regional training site or $2,500 a day for onsite training. 



Five studies reviewed by the WWC investigated the effects of 
Cognitive Tutor® Aigebra I. One study (Morgan & Ritter, 2002), 
which was reviewed in the original WWC report, was a random- 
ized controlled trial that met WWC evidence standards. One 
study (Shneyderman, 2001), which is new to this report, used a 
quasi-experimental design that met WWC evidence standards 
with reservations. The remaining three studies did not meet 
WWC evidence screens. 



Morgan and Ritter (2002) included 369 ninth-grade students 
in four suburban junior high schools in the Moore Independent 
School District in Oklahoma. Students in intervention classrooms 
used Cognitive Tutor® Algebra I, and students in the comparison 
group used McDougal Littell’s Heath Algebra 1, a traditional, 
teacher-directed curriculum. Students in the intervention and 
comparison groups attended the same schools and were taught 
by the same teachers. 
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Shneyderman (2001) included 439 ninth-grade students 
from six pubiic high schoois® in Miami-Dade County, Florida, 
during the 2000-01 school year. The intervention group used 
the Cognitive Tutor® Aigebra I textbook, ciassroom activities, 
and software. No information was provided on the comparison 
group other that these students did not use the Cognitive Tutor® 
Aigebra I textbook and software. Students in the intervention 
and comparison groups attended the same schools. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
smail or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Schemed . The extent of 
evidence takes into account the number of studies and the 
totai sample size across the studies that met WWC evidence 
standards with or without reservations.'* 

The WWC considers the extent of evidence for Cognitive 
Tutor® Aigebra I to be moderate to large for math achievement. 



Findings 

The WWC review of interventions for middie school math 
addresses student outcomes in the math achievement domain. 

Math achievement. Morgan and Ritter (2002) reported statisti- 
caily significant differences favoring Cognitive Tutor® Aigebra i 
students over comparison students on end-of-first semester and 
end-of-second semester math grades. The level of statistical 
significance for these outcomes was confirmed by the WWC. The 
study aiso reported no statisticaily significant differences between 
the groups on the Educational Testing Service (ETS) Algebra End- 
of-Course Assessment. The average effect size across ail three 
outcomes (as calculated by the WWC) was statistically significant. 

Shneyderman (2001) reported a statistically significant differ- 
ence between Cognitive Tutor® Aigebra i students and compari- 
son students on the ETS Algebra End-of-Course Assessment 
and no statistically significant differences on the Florida 
Comprehensive Assessment Test. These analyses included all 
the students in the sample. The WWC analysis, based on addi- 
tional information received from the study author that included 



only students who took both the pretest and posttest, found 
no statistically significant differences between the groups on 
either measure. Further, the average effect size across the two 
outcomes was neither statistically significant nor large enough 
to be considered substantively important according to the WWC 
standards (that is, at least 0.25). 

In sum, one study showed statistically significant positive 
effects, and one study showed indeterminate effects in the math 
achievement domain. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of 
the difference between participants in the intervention and 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme) . 



3. The ninth-grade sample of the Shneyderman (2001) study was reviewed for this report because the middle school math topic review focuses on grades 
6-9 regardless of setting (middle school, junior high school, or high school). For further details see the Middle School Math Protocol . 

4. The Extent of Evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept, external validity, such as students’ demographics and the types of 
settings in which studies took place, are not taken into account for the categorization. 

5. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Cognitive Tutor® Algebra /, corrections for clustering 
and multiple comparisons were needed. 
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The WWC found Cognitive 
Tutoi® Aigebra / to have 
potentially positive effects 
for math achievement 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see Technical Detaiis 
of WWC-Conducted Computations) . The improvement index rep- 
resents the difference between the percentiie rank of the average 
student in the intervention condition versus the percentiie rank 
of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is entirely based 
on the size of the effect, regardiess of the statisticai significance 
of the effect, the study design, or the analyses. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting results favorable to the intervention group. 



The average improvement index for math achievement is +8 
percentile points across the two studies, with a range of -1 to 
+16 percentile points across findings. 

Summary 

The WWC reviewed five studies on Cognitive Tutor® Algebra I. 
One of these studies met WWC evidence standards and another 
study met the WWC evidence standards with reservations; the 
remaining three studies did not meet WWC evidence screens. 
Based on these two studies, the WWC found potentially positive 
effects in the math achievement domain. The evidence pre- 
sented in this report may change as new research emerges. 



Rsferences Met WWC evidence standards 

Morgan, R, & Ritter, S. (2002). An experimental study of the 
effects of Cognitive Tutor Algebra I on student knowledge 
and attitude. Retrieved November 22, 2006, from http://www. 
carnegielearning.com/research/research_reports/morgan_ 
ritter_2002.pdf. 

Met WWC evidence standards with reservations 

Shneyderman, A. (2001, September). Evaluation of the Cogni- 
tive Tutor Algebra 1 Program. Unpublished manuscript. 
(Miami-Dade County Public Schools Office of Evaluation and 
Research, 1500 Biscayne Boulevard, Miami, FL 33132) 



Did not meet WWC evidence screens 

Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. 

A. (1997). Intelligent tutoring goes to school in the big city. 
International Journal of Artificial Intelligence in Education, 8, 
30-43.® 

Plano, G. S., Ramey, M., & Achilles, C. M. (2005, August). Impli- 
cations for student learning using a technology-based algebra 
program in a ninth-grade algebra course. Unpublished 
manuscript. (Mercer Island School District).^ 

Additional source: 

Plano, G. S. (2004). The effects of the Cognitive Tutor® Alge- 
bra on student attitudes and achievement in a 9th grade 
algebra course. Dissertation Abstracts International, 65(04), 
1291A. (UMI No. 3130130). 



6. Lack of evidence for baseline equivalence: the study, which used a quasi-experimental design, did not establish that the comparison group was equiva- 
lent to the intervention group at baseline. 

7. Lack of evidence for baseline equivalence: the study, which was reviewed as a quasi-experimental design, did not establish that the comparison group 
was equivalent to the intervention group at baseline. This study, which was designed as a regression discontinuity design, did not properly assign 
students at the cutoff grade. 
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Sarkis, H. (2004). Cognitive Tutor® Aigebra 1: Miami-Dade 
County Pubiic Schoois. Lighthouse Point, FL: The Reiiabiiity 
Group.® 



For more information about specific studies and WWC caicuiations, piease see the WWC Cognitive Tutor ® 
Algebra I Technicai Appendices . 



Cognitive Tutor® Algebra I 



May 29, 2007 



5 




Appendix 



Appendix A1.1 Study characteristics: Morgan & Ritter, 2002 (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Morgan, R, &Ritter, S. (2002). An experimental study of the effects of Cognitive Tutor Algebra i on student knowledge and attitude. Retrieved November 22, 2006, from 
http://www.carnegielearning.eom/research/research_reports/morgan_ritter_2002.pdf. 


Participants^ 


Three hundred and sixty-nine ninth-grade students participated in the part of the study that used random assignment at the section ievei and inciuded teachers who taught 
both conditions.^ The study authors eiiminated from the data set students who transferred from one ciassroom to another or who transferred in or out of the schooi (or district) 
during the course of the schooi year, resuiting in 342 students in 19 sections in the end-of-semester grades anaiysis and 255 students in 16 sections in the ETS totai scores 
anaiysis. About two-thirds of the students in the study were Caucasian, with the other third Asian, African-American, Hispanic, Native American, and other. None of the 
students were in speciai education. 


Setting 


Three suburban junior high schoois in the Moore independent Schooi District in Okiahoma. 


Intervention 


The intervention group used Cognitive Tutoi® Algebra 1. Students in this group spent three ciass periods each week in group activities and ciassroom discussions using the 
Cognitive Tutoi® Algebra /text and two ciass periods working on probiem-soiving skiiis with the Cognitive Tutor® Aigebra i software. The same teacher taught the intervention 
and comparison ciassroom in each of the three schoois. intervention group teachers taught Cognitive Tutor® tor the first time. 


Comparison 


The comparison group used Heath Aigebra i, pubiished by McDougai Litteii. The study does not provide further information on this curricuium. The same teachers taught the 
intervention and comparison ciassroom in each of the three schoois. Comparison group teachers had severai years of experience with the comparison curricuium. 


Primary outcomes 
and measurement 


This study used the Aigebra End-of-Course Assessment, deveioped by ETS, consisting of 25 muitipie-choice and 15 constructed-response questions. The other two outcomes 
were first semester grades and second semester grades. (See Appendix A2 for more detaiied descriptions of outcome measures.) 


Teacher training 


Aii intervention group teachers impiemented the curricuium for the first time. According to the study author, during the summer prior to the start of the program the teachers 
received the standard four-day course required for aii teachers beginning the program. 



1. The WWC requested and received from the second study author a description of the random assignment process and the resuits of an HLM analysis for students in classrooms that were 
randomly assigned within teachers. According to the study author, because of the availability of computers in the schools the study was limited to two teachers per school, and those teachers’ 
classes were randomly assigned to intervention or comparison. Therefore, this review focused on six teachers in three schools. 

2. Students from two additonal schools participated in the study. However, because random assignment within teachers did not occur in these schools, these data were not reviewed for rating 
purposes. 
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Appendix A1.2 Study characteristics: Shneyderman, 2001 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Shneyderman, A. (2001, September). Evaluation of the Cognitive Tutor Algebra 1 Program. Unpublished manuscript. (Miami-Dade County Public Schools Office of Evaluation 
and Research, 1500 Biscayne Boulevard, Miami, FL 33132.) 


Participants' 


The quasi-experimentai study included 439 ninth-grade students (191 in the intervention and 248 in the comparison group) who were enrolled in algebra 1 during the 
2000-01 school year.' In each of the participating schools, two teachers of predominantly regular education students were randomly selected. One class per teacher was 
included in the intervention group. The comparison group comprised an equal number of students within the same schools. About 47% of the students were female, and 54% 
of the students participated in the free or reduced-price lunch program. About 9% of the students in the intervention group and 20% of the students in the comparison group 
were English language learners. 


Setting 


The participating students were from six Miami-Dade County public senior high schools that used Cognitive Tutoi® Aigebra i dunnQ the 2000-01 school year. These schools 
were selected from the nine schools in the district that implemented the curriculum based on the availability of computer labs for student use at the time of the study, because 
the curriculum involves extensive computer use. 


Intervention 


Students spent about 40% of their mathematics instructional time using the Cognitive Tutoi® Aigebra / software in their algebra classes during the 2000-01 school year. 
Students spent the other 60% of their mathematics instructional time working in small groups in the classroom setting using the Cognitive Tutoi® Algebra /textbook. 


Comparison 


Comparison students took algebra 1 but did not use the Cognitive Tutor® Aigebra t software and textbook. The author does not report what curricula the comparison students used. 


Primary outcomes 
and measurement 


The primary outcome measures were the norm referenced component of the Florida Comprehensive Assessment Test (FCAT-NRT) and the ETS Algebra End-of-Course 
Assessment (EOCT).^ (See Appendix A2 for detailed descriptions of outcome measures.) 


Teacher training 


The study did not provide information on teacher training. 



1. The WWC requested and received from the study author sample size, means, and standard deviations for a subsample of ninth-grade students who received both the pretests and the posttests. 
Results for ninth-grade students with no pretest scores were not included in this review because this subsample did not meet the WWC evidence standards for quasi-experimentai designs. 
Results for other students included in this study were not reviewed because they attended higher grade levels (see the Middle School Math Protocol for further detail on inclusion criteria). 

2. The study also reported on end-of-year algebra I passing rates and final grades, but these analyses did not take into account baseline differences and so were not reviewed by the WWC. 
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Appendix A2 Study characteristics: Shneyderman, 2001 (quasi-experimentai design) 



Outcome measure 


Description 


ETS Algebra End-of-Course 
Assessment (EOCT) 


The ETS Algebra End-of-Course Assessment (EOCT) included 25 multiple-choice and 15 constructed-response (or performance) items. Each of fhe performance ifems 
contained four parts that progressively increased in difficulty level (as cited in Morgan and Ritter, 2002; Shneyderman, 2001). 


Florida Comprehensive 
Assessment Test — Norm 
Reference Test (FCAT-NRT) 


The mathematics subtest of fhe state standardized test, the ECAT-NRT consisted of 48 multiple-choice questions in 10 areas, ranging from problem solving to precalculus. 
Test results were reported as scale scores, mean number of multiple-choice items correct, and mean algebra multiple-choice items correct (as cited in Shneyderman, 2001). 


Math achievement grades 


End-of-semester and end-of-year grades in mathematics (as cited in Morgan and Ritter, 2002). 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students)^ 


Cognitive Tutoi^ Comparison 

group group 


Mean difference’’ 
(Cognitive 
Tutor® - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index^ 



ETS Algebra End-of-Course 
Assessment 


Grade 9 


Morgan & Ritter, 2002 (randomized controlled trial)® 

16/255 16.92 15.28 2.13 

(5.82) (5.33) 


0.38 


ns 


+15 


Math achievement grades (end 
of first semester) 


Grade 9 


19/342 


3.06 

(0.99) 


2.76 

(1.16) 


0.45 


0.42 


Statistically 

significant 


+16 


Math achievement grades (end 
of second semester) 


Grade 9 


19/342 


2.74 

(1.11) 


2.38 

(1.29) 


0.44 


0.37 


Statistically 

significant 


+14 


Average® for math achievement (Morgan & Ritter, 2002) 








0.39 


ns 


+15 






Shneyderman, 2001 (quasi-experimental design)® 








ETS Algebra End-of- 


Grade 9 


24/360 


10.27 


9.90 


0.37 


0.07 


ns 


+3 


Course Assessment 






(5.14) 


(4.79) 










ECAT-NRT 


Grade 9 


24/439 


681.85 


682.39 


-0.54 


-0.02 


ns 


-1 








(29.71) 


(28.41) 










Average® for math achievement (Shneyderman, 2001) 








0.03 


ns 


+1 


Domain average® for math achievement across all studies 








0.21 


na 


+8 



ns = not statistically significant 

na = not applicable 

1. This appendix reports finding considered for the effectiveness rating and the average improvement indices. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For Morgan 
and Ritter (2002) and Shneyderman (2001), means and standard deviations were received from fhe study authors; the WWC further estimated the combined standard deviation across all teachers, taking into account the number of 
studenfs perfeacher. In Morgan and Riffer (2002), the intervention group mean is the comparison group mean plus the mean difference. 

3. Sample size information presented here was requested by the WWC and received from the study authors. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of fhe effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting results favorable to the intervention group. 



(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain (continued) 



8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Shneyderman (2001), corrections for 
clustering and multiple comparisons were needed. In the case of Morgan and Ritter (2002), corrections for multiple comparisons were needed; corrections for clustering In this study were not needed because the study showed statisti- 
cally significant effects for two outcomes based on analysis that takes clustering Into account. The significance levels may differ from those reported In the original studies. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average Improvement Indices are calculated from the average effect size. 
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Appendix A4 Cognitive Tutoi® Aigebra i rating for the math achievement domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negativeJ 
For the outcome domain of math achievement, the WWC rated Cognitive Tutor® Aigebra I as having potentialiy positive effects. It did not meet the criteria for positive 
effects because only one study showed statistically significant positive effects. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, 
and negative effects) were not considered because Cognitive Tutor® Algebra I was assigned the highest applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study of Cognitive Tutor® Algebra I showed statistically significant positive effects. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. One study of Cognitive Tutor® Algebra I showed an indeterminate effect and no studies showed statistically significant or substantively 
important negative effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. One study of Cognitive Tutor® Algebra I showed statistically significant positive effects. The second study of Cognitive Tutor® Algebra I 
showed indeterminate effects. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence^ 


Math achievement 


2 


9 




781 


Moderate to large 



1 . 



A rating of “moderate to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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